Platform for automated scoring of scientific visual models

ABSTRACT

Systems and methods are provided for processing a drawing in a modeling prototype. A data structure associated with a visual model is accessed. The visual model is analyzed to extract construct-relevant features, where the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn. The visual model is analyzed to generate a statistical model, where the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs. An automated scoring is determined based on the construct-relevant features and the statistical model, where the automated scoring is stored in a computer readable medium. and is outputted for display, transmitted across a computer network, or printed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/645,485, filed Mar. 20, 2018, the entirety of which is hereinincorporated by reference.

TECHNICAL FIELD

The technology described herein relates to automated scoring ofscientific visual models and more particularly to evaluation of asubject's understanding of scientific concepts.

BACKGROUND

Scientists use models to represent their understanding of a phenomenon,including facilitating the development of research questions,explanations, predictions, and communications with others. Intuitively,evaluation of visual models created by students to represent theirscientific understanding is a promising way to assess application ofknowledge acquired. However, scaling up evaluation of such visual modelsrequire standardization of a feature framework to disentangle artisticelements from modeling skills to ensure fairness in the scoring process.Further, human scoring of these visual models is often inconsistent andresults in unfair assessments. The technology described herein usesconstruct-relevant features to build scoring models that areinterpretable and deployable in a large-scale setting for automatedscoring.

SUMMARY

Systems and methods are provided for processing a drawing in a modelingprototype. A data structure associated with a visual model is accessed.The visual model is analyzed to extract construct-relevant features,where the construct-relevant features are extracted using a drawingobject by identifying visual attributes of the visual model andpopulating a data structure for each object drawn. The visual model isanalyzed to generate a statistical model, where the statistical model isgenerated using a multidimensional scoring rubric by targeting differentconstructs which compositely estimate learning progression levels,wherein the statistical model is based on features that are principallyaligned with one or more of the constructs. An automated scoring isdetermined based on the construct-relevant features and the statisticalmodel, where the automated scoring is stored in a computer readablemedium and is outputted for display on a graphical user interface,transmitted across a computer network, or printed.

As another example, a system for processing a drawing in a modelingprototype includes one or data processors and a computer-readable mediumencoded with instructions for commanding the one or more processors toexecute steps. In the steps, a data structure associated with a visualmodel is accessed. The visual model is analyzed to extractconstruct-relevant features, where the construct-relevant features areextracted using a drawing object by identifying visual attributes of thevisual model and populating a data structure for each object drawn. Thevisual model is analyzed to generate a statistical model, where thestatistical model is generated using a multidimensional scoring rubricby targeting different constructs which compositely estimate learningprogression levels, wherein the statistical model is based on featuresthat are principally aligned with one or more of the constructs. Anautomated scoring is determined based on the construct-relevant featuresand the statistical model, where the automated scoring is stored in acomputer readable medium and is outputted for display on a graphicaluser interface, transmitted across a computer network, or printed.

As a further example, a computer-readable medium is encoded withinstructions for commanding one or more data processors to execute amethod for processing a drawing in a modeling prototype. In the method,a data structure associated with a visual model is accessed. The visualmodel is analyzed to extract construct-relevant features, where theconstruct-relevant features are extracted using a drawing object byidentifying visual attributes of the visual model and populating a datastructure for each object drawn. The visual model is analyzed togenerate a statistical model, where the statistical model is generatedusing a multidimensional scoring rubric by targeting differentconstructs which compositely estimate learning progression levels,wherein the statistical model is based on features that are principallyaligned with one or more of the constructs. An automated scoring isdetermined based on the construct-relevant features and the statisticalmodel, where the automated scoring is stored in a computer readablemedium and is outputted for display on a graphical user interface,transmitted across a computer network, or printed.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram depicting a computer-implemented system forprocessing a drawing in a modeling prototype.

FIG. 2 is a block diagram depicting a computer-implemented system forprocessing a drawing to generate a visual model score that istransmitted across a computer network or displayed on a graphical userinterface.

FIG. 3 is a diagram depicting a computer-implemented extraction modulefor extracting construct-relevant features from a visual model.

FIG. 4 is a diagram depicting a computer-implemented model trainer forgenerating a statistical model from a visual model.

FIG. 5 is a diagram depicting a computer-implemented prediction modulefor generating a score prediction from a visual model.

FIG. 6 is a diagram depicting a computer-implemented scoring module forextracting a scoring rubric from a visual model.

FIG. 7 is a diagram depicting a computer-implemented learningprogression module for extracting a learning progression level from avisual model.

FIG. 8 is a flow diagram depicting a processor-implemented method forprocessing a drawing in a modeling prototype.

FIGS. 9A, 9B, and 9C depict example systems for implementing theapproaches described herein for automatically scoring a visual model ina modeling prototype.

FIG. 10 shows examples of models generated using a computer system bystudents to illustrate scientific models of pure water (first row) andocean water (second row).

FIG. 11 is a table that reflects marginal correlations of individualfeature against human scoring dimensions.

FIG. 12 is a table that shows marginal correlations based on thetraining partition for each feature computed against each of the scoringdimensions.

DETAILED DESCRIPTION

Systems and methods as described herein automatically score visualmodels of scientific concepts drawn by students. Visual models ofscientific concepts drawn by students affords expanded opportunities forstudents to show their understanding of subject matter, but alsointroduces other elements characterized by artistic creativity andcomplexity. A standardized feature extraction framework for automatedscoring of visual models applied to a scientific concept is provided.This disclosure is provided in the context of visual models representingconcepts related to the concept of Matter (i.e., the substance orsubstances of which any physical object consists or is composed) and itsbehavior. The techniques herein are equally applicable to mathematics,science, and other concepts. This framework disentangles the interactionbetween student scientific modeling skill from their art skill ofrepresenting real objects and provides for a fair and valid way toassess understanding of subjects Matter by students.

In one example, preliminary evaluation of models constructed by thestandardized feature set achieved accuracy of up to 88%, and was able toexplain approximately 46% of the variance in learning progression scoresobtained by students.

Assessment experts have noted that new reforms in science educationrequire innovative assessments to probe multiple dimensions of scienceknowledge such as core ideas and science practices. Policy experts andscience education researchers have also called for the use of learningprogressions (LPs) to guide assessment development in order to developmore diagnostic tools of knowledge acquired by students and the abilityto inform instruction.

The Next Generation Science Standards (NGSS) explicitly identifiedmodeling as one central and valued practice, and modeling is alsoidentified as an important practice in mathematics. The visual modelsconstructed by students can serve as rich vehicles of information foreducators interested in supporting and assessing what students know andcan do in science.

In one example, a new computer-based science assessment aligned with theNGSS and a learning progression (LP) was developed in order to determineif we better measures of student learning in science could be built. Acore disciplinary idea (i.e. Matter) and a central practice (i.e.developing and using models) were selected as the target constructs forthe assessment prototype that addressed the multidimensional features ofscience learning.

In one example, hand-written drawn models by students were employed as arich source of evidence to explore what they know about the structureand behavior of Matter, and these were used to construct an LP forMatter. However, challenges remained with regards to a large scaleassessment of drawings by students. One obvious hurdle was the expensivelabor costs associated with human scoring of such drawings at scale.

To develop the various computer evaluation system described herein, waysto automate the scoring process to assess object-based drawingsgenerated by students were explored. A fair and valid assessment ofdrawings must disentangle the interaction between the scientificmodeling skills and the art skills of representing real objects bystudents. The findings resulted in an updated knowledge of cutting-edgeautomated scoring methods that can be applied to score student-generatedmodels, and also inform process of designing modeling prototypes tomeasure integrated science competency.

Human scoring of visual models are based on a developed scoring rubricthat are mapped with the learning progression for Matter. The scoringscheme for this example includes four dimensions that address the scale(S), material identity (MI), behavior (B), and distribution (D) ofparticles concerned with Matter LP. For a given visual model, the scaledimension measures understanding of hierarchical composition of Matterbeginning with the smallest units e.g. nanoscopic particles. Thematerial identity dimension examines the anticipated number of particleidentities present. The behavior dimension examines if/how particlemovement is represented. The distribution dimension examines positionsof individual particles and space between them in liquid Matter state.In order to climb the learning progression, a student must exhibit amastery of levels anticipated in each dimension. For example,progression from LP-3 to LP-4 requires a mastery of level 3 in scaledimension and a minimum of level 2 in behavior dimension.

In one example, a table showed a mapping between Learning Progression(LP) levels and levels in each human scoring dimension. For a given LPlevel, indication of a ‘X’ meant the minimum level that must be masteredin that associated dimension.

S MI B MI 0 1 2 3 0 1 2 0 1 2 3 0 1 2 LP-1 X X X X X X LP-2 X X X X X XLP-3 X X X X X X X LP-4 X X X X X X X LP-5 X X X X

In an example, visual models were collected through a pilot study thatexplored the implementation of a formative assessment prototype in twoscience classroom settings. In both classroom settings, teachers usedthe prototype assessment task to help students learn about the core ideaof Matter. The formative assessment task was delivered online andstudents worked in pairs to input responses due to lack of access totechnology in both classroom settings. Both teachers implemented theassessment task during a relevant unit of science instruction. Beforeimplementing the task, both teachers received a one-day professionaldevelopment training on strategies for using formative assessment, theunderlying science competency model, and the Matter LPs, the formativeassessment task, and relevant supporting materials for implementation(e.g., the teacher handbook and scoring rubrics).

In an example, students were asked to draw and refine models of purewater and ocean water. Modeling items involved the use of acomputer-based drawing tool in which students used a virtual pen orselected from a pool of predefined objects, including abstract objects(e.g., circles, squares, etc.) and concrete representations (e.g., fish,rocks, water drops, sand), to allow students to express their idea ofstructure of Matter. The drawing tool also allowed students to changethe size or color of selected objects, add arrows to represent motion,and label objects. In total, about 1123 student drawings of particlemodels of Matter were collected, about half of which were human coded bytwo individual raters. FIG. 10 shows examples of models generated usinga computer system by students to illustrate scientific models of purewater (first row) and ocean water (second row). Students had at theirdisposal to use micro-objects, macro-objects, labels, and pictures,where micro-objects={circle, square, triangle, diamond},macro-objects={fish, water drops, water steam, algae, salt, etc.}.Directional arrows could also be used to illustrate behavioral patternsof water molecules.

In the example, it was found most students' models were scored at a lowlevel i.e. they held macroscopic conceptions of Matter, and only a smallfraction of students' models were scored at high levels i.e. they heldbeginning or well developed nanoscopic conceptions of Matter. Onaverage, 81.5% of student responses were scored at level 1 or 2, while17.5% were scored at level 3 or 4 (with only 2.5% at level 4).Additionally, it was found that many students held mixed model 8% ofpure water models were mixed model with overt or less overt macrorepresentations; 29% of ocean water models were mixed model with overtor less overt macro representations.

In the example, students exhibited widely contrasting approaches tovisual modeling even within the same concept (pure water or oceanwater). Freedom in choice of modeling tools was necessary to elicitresponses corresponding to true underlying understanding of concepts, atthe same time affording a richness of expression in conveying thatunderstanding. However, variations in the choice, color, scale, positionand motion of objects suggested standardization of an evaluationframework was not only necessary for a fair and valid assessment, butalso required for building models for automated scoring of these visualmodels.

In an embodiment, two key steps were adopted by automated scoringapproaches for high-stakes learning assessment. First,construct-relevant features were extracted from the data that werecorrelated to human ratings on the scoring dimensions. Second,statistical models were built using these features to automate thescoring process. Given the visual nature of the dataset, the embodimentfocused on a unimodal approach.

In one example, verbal description of models such as “my model showsrain drops and blue squares because the square represents the ocean andthe drops represent the rain” were also available for analysis usingnatural language processing techniques.

Regarding the drawing object, in an embodiment, each visual model was aresponse by a student elicited in the form of a drawing in the computerimplemented modeling prototype system. The visual attributes of thedrawing were located in the corresponding Javascript Object Notation(JSON) file that was self-descriptive. For each object drawn, JSONencoded its type, color (RGB with an alpha channel for specificopacity), text, X-Y coordinates, height, width and rotation in degrees.Each object drawn was one of macro-objects, micro-objects, label, orarrow.

Regarding the inserted textual description, in one example, in the caseof label, the student could elect to insert a textual description.

In one example, data cleaning was necessary prior to the featureextraction. Of the 1123 JSON files generated, 453 with meaningfulcontent were obtained after the cleaning step where (1) empty JSON fileswere removed (2) JSON files that failed the parser were dropped.Additionally, for each of the 453 JSON files that remained, aheuristics-based preprocessing step was performed to ensure theintegrity of the finalized model features.

Regarding the background micro-object, in an example, for any pair ofoverlapping micro-objects, the one that is used entirely as a backgroundbelow the other micro-object was removed, as such an object added nosemantics to overall understanding by students. This preprocessingensured that each remaining micro-object would not be obscured by anyother micro objects.

The multidimensional scoring rubric target different constructs whichcompositely estimate the learning progression level of a student. In anembodiment, two categories of features, counting-based features andspatial-based features, were hypothesized, each principally aligned withone or more of the constructs to ensure coverage in the scoring process.

Regarding the counting-based features, a basic understanding of eachscientific concept during visual modeling rested on knowledge of thenumber of type of particles present. This was 1 for pure water model and2 for ocean water model (salt and water particles), and a uniqueparticle identity could be specified by color, type or a combination ofboth using micro-objects. Deviation from these expected counts indicateda significant gap in material identity awareness. Likewise,macro-objects such as fishes and water drops when overused relative tomicro-objects signaled shallow understanding in the scale dimension.Behavioral wise, arrows indicated direction of movement of particles andtheir lengths were used to gauge velocity of such movements.

Regarding the spatial-based features, two aspects that specificallytarget the distributive property of particles were worth concern. Toestimate spatial tightness and looseness, the k-Nearest Neighbor (k-NN)algorithm was adopted to compute inter-particle distances. k=3 was usedfor a local approximation of proximity, and k=10 was used for a moreglobal approximation. For a given visual model, the dispersion featurecomputed the number of particles per unit area per particle type, andaveraging over all particle types. A larger dispersion value wassuggestive of the same number of particles drawn over a larger canvasarea in the visual model.

In one example, the table in FIG. 11 reflected marginal correlations ofindividual feature against human scoring dimensions. The columnsincluded: S=Scale; MI=Material Identity; B=Behavior; D=Distribution;LP=Learning Progression. Except for macro-object types, all otherfeatures were based on micro-objects. Magnitude-wise, the largestcorrelation in each feature category per dimension was underlined, whilethe largest correlation overall within a dimension was in bold.

Regarding the visual model score, in one example, given the numericlabels assigned to each LP level, it was possible to formulate the scoreprediction process as a supervised task of regression or classificationusing learners with matured statistical properties and explainableoutputs, which were recommended for high-stakes assessment tasks.

In the example, RSMTool was used for building and evaluating thelearners that were potentially deployable as the automated scoringsystem. RSMTool is a python package which automates and combines in asingle pipeline multiple analyses that are commonly conducted whenbuilding and evaluating automated scoring models.

In an example, the 263 visual models represented by JSONs were shuffledrandomly and then split into 237 JSONs for training and 26 JSONs forevaluation respectively. Of these, 70-10 train-evaluation split was forpure water model, while 167-16 split was for ocean water model.Counting-based and spatial-based features were extracted for a total of10 features per visual model. Specifically, log transformations wereapplied to spatial features for data smoothing. Marginal correlationsbased on the training partition for each feature computed against eachof the scoring dimensions, as well as LP, were shown in the table inFIG. 12.

In the example, given that LP level prediction was the ultimate goal forlearning assessments on Matter, RSMTool was used to build a linearregression model using all available features and compute associatedstatistics. Additionally, the task was framed in a classificationsetting using several classification models noted for theireffectiveness. The overall accuracies were reported the previous table.

In the example, a number of observations regarding correlations weremade by referring to the table. First, the consistent negativecorrelations of the macro-object types feature across the differentdimensions indicated that understanding levels of Matter were lesssophisticated when students focused on drawing more macroscopic objectsrather than explaining the microscopic or nanoscopic aspects of Matter.This was particularly evident of modeling the scale dimension. Second,the number of arrows, their direction and randomness almost exclusivelyaccounted for showing understanding of the behavior of particles inMatter by students when compared to other features. Third, it wasobserved that all spatial-based features bore promising correlations(|r|˜0.50) in modeling the distribution dimension. Specifically, thedispersion feature stood out among all features in it consistencymodeling all except for the behavior dimension. The EIC deviation wasthe only one that was engineered to target concept-specific visualmodel, where its value was dependent on whether the model was pure waterand ocean water. Expectedly, this feature had a correlation of −0.393for the material identity dimension, which indicated students would bepenalized for deviating away from the expected number of identitiesanticipated.

In the example, after controlling for all other variables, analysissuggested that micro-object types, macro-object types, arrows anddispersion features were the most correlated with LP with partialcorrelations of 0.18, −0.25, 0.26 and −0.20 respectively. This findingcalled for students to target an all-around visual modeling approachthat focus on microscopic aspects of Matter, its behavior and takingadvantage of the entire canvas while doing so.

In an example, a comparative evaluation of models were built using bothregressors and classifiers, as shown in the following table. In thetable, learners with the same accuracy numbers displayed significantlydifferent confusion matrices, indicating different strengths at modelingdifferent LP levels. Though LP levels could be classified numerically,prediction accuracy was consistently better using regression-basedlearners of which the maximum was achieved at 0.88. An adjusted R² of0.46 was reported in liner regression model, which suggests thatapproximately half the variance in LP level differences could beaccounted for by a simple model based on linear regression model usingthe feature set proposed.

Learner Accuracy Linear Regression 0.81 Decision Tree Regression 0.88Random Forest Regression 0.88 Logistic Regression Classifier 0.65Decision Tree Classifier 0.65 Random Forest Classifier 0.65

In an example, it was suggested that further feature engineeringassociated with the model scores with a broad-based sample covering thefull range of the score levels (or learning progression levels) wouldproduce convincing results. It was suggested that related efforts wouldbe targeted toward constructs with high partial correlations to LP,namely the behavior and distribution dimension.

FIG. 1 is a block diagram depicting a computer-implemented system forprocessing a drawing in a modeling prototype. A visual model scoringengine 102 accesses a data structure 104 associated with a drawing. Thescoring engine 102 may access one or more schemes 106 that contain datato assist in automatically extracting metrics from the drawing 104, suchas described above. For example, a scoring scheme 106 may be accessed tofacilitate generation of a scoring rubric, a linear regression model 108may be accessed to facilitate generation of a learning progressionlevel, the visual model scoring engine, in one example, determines oneor more of construct-relevant features, a statistical model, a scoringrubric, an integrity of the finalized model features, and a backgroundmicro-object based on the features extracted from the drawing 104. Thosedetermined features are output from the engine 102 as a drawing score110.

FIG. 2 is a block diagram depicting a computer-implemented system forprocessing a drawing to generate a visual model score that istransmitted across a computer network or displayed on a graphical userinterface. In the example of FIG. 2, the visual model scoring engine 202includes an extraction module 206 that receives a data structureassociated with a visual drawing 204. In the example of FIG. 2, thevisual model scoring engine 202 includes a model trainer 210 thatreceives a data structure associated with a training drawing 204. In theexample of FIG. 2, the visual model scoring engine 202 includes anautomated scoring model that receives the output from the extractionmodule 206 and the output from the model trainer 210 to generate avisual model score. In another example, the engine 202 includes aprediction module that generates a score prediction. In another example,the engine 202 includes a scoring module that generates a scoringrubric. In another example, the engine 202 includes a learningprogression module that generates a learning progression level.

FIG. 3 is a diagram depicting a computer-implemented extraction modulefor extracting construct-relevant features from a visual model. Theconstruct-relevant features 314 are extracted from the visual model 302.The extraction module at 304 extracts construct-relevant features at306. The construct-relevant features 314 are extracted using a drawingobject 310. The drawing object is extracted by identifying visualattributes of the visual model at 308 and at 312 populating a datastructure for each object drawn.

FIG. 4 is a diagram depicting a computer-implemented model trainer forgenerating a statistical model from a visual model. The statisticalmodel 414 is generated from the visual model 402. The model trainer at404 generates a statistical model at 406. The statistical model isgenerated using a multidimensional scoring rubric 410. The statisticalmodel 414 is generated by targeting different constructs at 408 and at412 estimating learning progression levels. In one example, thestatistical model is based on features that are principally aligned withone or more of the constructs.

FIG. 5 is a diagram depicting a computer-implemented prediction modulefor generating a score prediction from a visual model. In the example, ascore prediction 502 is determined based on learners 504 alone or incombination with other metrics automatically extracted from the visualmodel 506. The learners 504 are determined by the prediction module 508at 510 by identifying the learners in the visual model 506 with maturedstatistical properties and explainable outputs using the supervised taskof regression or classification 512. In one example, the scoreprediction 502 is combined with construct-relevant features and astatistical model to determine a visual model score.

FIG. 6 is a diagram depicting a computer-implemented scoring module forextracting a scoring rubric from a visual model. The scoring module 602accesses a scoring scheme 604 at 466 that identifies dimensions ofparticles concerned with a scientific concept. At 608, the module 602extracts the dimensions from the visual model 610 and determines whethereach extracted dimension is in the scoring scheme. The scoring rubric612 is generated at 614 based on dimensions that are located in thescoring scheme 404.

FIG. 7 is a diagram depicting a computer-implemented learningprogression module for extracting a learning progression level from avisual model. The learning progression module 702 accesses a scoringscheme 704 at 706 that automates and combines multiple analyses in asingle pipeline. At 708, the module 702 builds and evaluates learnersfrom the visual model 710. The learning progression level 712 isgenerated at 714 based on multiple analyses located in the linearregression model 704.

FIG. 8 is a flow diagram depicting a processor-implemented method forprocessing a drawing in a modeling prototype. A data structureassociated with a visual model is accessed at 802. The visual model isanalyzed at 804 to extract construct-relevant features, where theconstruct-relevant features are extracted using a drawing object byidentifying visual attributes of the visual model and populating a datastructure for each object drawn. The visual model is analyzed at 806 togenerate a statistical model, where the statistical model is generatedusing a multidimensional scoring rubric by targeting differentconstructs which compositely estimate learning progression levels,wherein the statistical model is based on features that are principallyaligned with one or more of the constructs. A visual model score isdetermined at 808 on the construct-relevant features and the statisticalmodel, where the visual model score is stored in a computer readablemedium and is outputted for display on a graphical user interface,transmitted across a computer network, or printed.

FIGS. 9A, 9B, and 9C depict example systems for implementing theapproaches described herein for automatically scoring a visual model ina modeling prototype. For example, FIG. 9A depicts an exemplary system900 that includes a standalone computer architecture where a processingsystem 902 (e.g., one or more computer processors located in a givencomputer or in multiple computers that may be separate and distinct fromone another) includes a computer-implemented visual model scoring engine904 being executed on the processing system 902. The processing system902 has access to a computer-readable memory 907 in addition to one ormore data stores 908. The one or more data stores 908 may include ascoring scheme 910 as well as a linear regression model 912. Theprocessing system 902 may be a distributed parallel computingenvironment, which may be used to handle very large-scale data sets.

FIG. 9B depicts a system 920 that includes a client-server architecture.One or more user PCs 922 access one or more servers 924 running a visualmodel scoring engine 937 on a processing system 927 via one or morenetworks 928. The one or more servers 924 may access a computer-readablememory 930 as well as one or more data stores 932. The one or more datastores 932 may include a scoring scheme 934 as well as a linearregression model 938.

FIG. 9C shows a block diagram of exemplary hardware for a standalonecomputer architecture 950, such as the architecture depicted in FIG. 9Athat may be used to include and/or implement the program instructions ofsystem embodiments of the present disclosure. A bus 952 may serve as theinformation highway interconnecting the other illustrated components ofthe hardware. A processing system 954 labeled CPU (central processingunit) (e.g., one or more computer processors at a given computer or atmultiple computers), may perform calculations and logic operationsrequired to execute a program. A non-transitory processor-readablestorage medium, such as read only memory (ROM) 958 and random accessmemory (RAM) 959, may be in communication with the processing system 954and may include one or more programming instructions for performing themethod of automatically scoring a visual model in a modeling prototype.Optionally, program instructions may be stored on a non-transitorycomputer-readable storage medium such as a magnetic disk, optical disk,recordable memory device, flash memory, or other physical storagemedium.

In FIGS. 9A, 9B, and 9C, computer readable memories 907, 930, 958, 959or data stores 908, 932, 983, 984, 988 may include one or more datastructures for storing and associating various data used in the examplesystems automatically scoring a visual model in a modeling prototype.For example, a data structure stored in any of the aforementionedlocations may be used to store data from XML files, initial parameters,and/or data for other variables described herein. A disk controller 990interfaces one or more optional disk drives to the system bus 952. Thesedisk drives may be external or internal floppy disk drives such as 983,external or internal CD-ROM, CD-R, CD-RW or DVD drives such as 984, orexternal or internal hard drives 985. As indicated previously, thesevarious disk drives and disk controllers are optional devices.

Each of the element managers, real-time data buffer, conveyors, fileinput processor, database index shared access memory loader, referencedata buffer and data managers may include a software application storedin one or more of the disk drives connected to the disk controller 990,the ROM 958 and/or the RAM 959. The processor 954 may access one or morecomponents as required.

A display interface 987 may permit information from the bus 952 to bedisplayed on a display 980 in audio, graphic, or alphanumeric format.Communication with external devices may optionally occur using variouscommunication ports 982.

In addition to these computer-type components, the hardware may alsoinclude data input devices, such as a keyboard 979, or other inputdevice 981, such as a microphone, remote control, pointer, mouse and/orjoystick.

Additionally, the methods and systems described herein may beimplemented on many different types of processing devices by programcode comprising program instructions that are executable by the deviceprocessing subsystem. The software program instructions may includesource code, object code, machine code, or any other stored data that isoperable to cause a processing system to perform the methods andoperations described herein and may be provided in any suitable languagesuch as C, C++, JAVA, for example, or any other suitable programminglanguage. Other implementations may also be used, however, such asfirmware or even appropriately designed hardware configured to carry outthe methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, datainput, data output, intermediate data results, final data results, etc.)may be stored and implemented in one or more different types ofcomputer-implemented data stores, such as different types of storagedevices and programming constructs (e.g., RAM, ROM, Flash memory, flatfiles, databases, programming data structures, programming variables,IF-THEN (or similar type) statement constructs, etc.). It is noted thatdata structures describe formats for use in organizing and storing datain databases, programs, memory, or other computer-readable media for useby a computer program.

The computer components, software modules, functions, data stores anddata structures described herein may be connected directly or indirectlyto each other in order to allow the flow of data needed for theiroperations. It is also noted that a module or processor includes but isnot limited to a unit of code that performs a software operation, andcan be implemented for example as a subroutine unit of code, or as asoftware function unit of code, or as an object (as in anobject-oriented paradigm), or as an applet, or in a computer scriptlanguage, or as another type of computer code. The software componentsand/or functionality may be located on a single computer or distributedacross multiple computers depending upon the situation at hand.

While the disclosure has been described in detail and with reference tospecific embodiments thereof, it will be apparent to one skilled in theart that various changes and modifications can be made therein withoutdeparting from the spirit and scope of the embodiments. Thus, it isintended that the present disclosure cover the modifications andvariations of this disclosure provided they come within the scope of theappended claims and their equivalents.

It is claimed:
 1. A processor implemented method of processing a drawing in a modeling prototype, the method comprising: accessing a data structure associated with a visual model; analyzing the visual model to extract construct-relevant features, wherein the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn; analyzing the visual model and associated scores to generate a statistical model, wherein the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs; determining an automated scoring based on the construct-relevant features and the statistical model, wherein the automated scoring is stored in a computer readable medium and is outputted for display on a graphical user interface, transmitted across a computer network, or printed.
 2. The method of claim 1, further comprising: analyzing the visual model to generate a score prediction, wherein the score prediction is generated using learners with matured statistical properties and explainable outputs by a supervised task of regression or classification.
 3. The method of claim 2, further comprising: determining a visual model score based on the construct-relevant features, the statistical model, and the score prediction.
 4. The method of claim 1, further comprising determining a scoring rubric by accessing a scoring scheme that identifies dimensions of particles concerned with a scientific concept, wherein the scoring rubric is based on dimensions that are located in the scoring scheme.
 5. The method of claim 4, wherein the scoring rubric is further based on a proportion of dimensions found in the scoring scheme that address the scale, material identity, behavior, and distribution of particles concerned with the scientific concept.
 6. The method of claim 1, wherein the drawing object is further determined using a file that is self-descriptive based on type, color, text, X-Y coordinates, height, width and rotation.
 7. The method of claim 1, wherein the construct-relevant features are determined based on a file that is not empty and did not fail the parser.
 8. The method of claim 7, further comprising determining the integrity of the finalized model features by performing a heuristics-based preprocessing step.
 9. The method of claim 1, wherein the visual attributes are further determined based on one of or a plurality of macro-objects, micro-objects, label, or arrow.
 10. The method of claim 9, further comprising determining the label using an inserted textual description.
 11. The method of claim 9, further comprising determining a background micro-object by determining a pair of overlapping micro-objects and removing the one used entirely as a background below the other micro-object.
 12. The method of claim 1, wherein the statistical model is further determined based on counting-based features and spatial-based features.
 13. The method of claim 1, further comprising determining a learning progression level by accessing a linear regression model that automates and combines multiple analyses in a single pipeline, wherein the multiple analyses are determined by building and evaluating learners.
 14. The method of claim 1, wherein the drawing object is generated using a computer-based drawing tool by determining input from a virtual pen or selected from a pool of predefined objects.
 15. A processor implemented system for processing a drawing in a modeling prototype, comprising: one or more data processors; a computer-readable medium encoded with instructions for commanding the one or more data processors to execute steps of a process, the steps including: accessing a data structure associated with a visual model; analyzing the visual model to extract construct-relevant features, wherein the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn; analyzing the visual model to generate a statistical model, wherein the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs; determining an automated scoring based on the construct-relevant features and the statistical model, wherein the automated scoring is stored in a computer readable medium and is outputted for display on a graphical user interface, transmitted across a computer network, or printed.
 16. The system of claim 15, the steps further comprising: analyzing the visual model to generate a score prediction, wherein the score prediction is generated using learners with matured statistical properties and explainable outputs by a supervised task of regression or classification.
 17. The system of claim 16, the steps further comprising: determining a visual model score based on the construct-relevant features, the statistical model, and the score prediction.
 18. The system of claim 15, the steps further comprising determining a scoring rubric by accessing a scoring scheme that identifies dimensions of particles concerned with a scientific concept, wherein the scoring rubric is based on dimensions that are located in the scoring scheme.
 19. The system of claim 18, wherein the scoring rubric is further based on a proportion of dimensions found in the scoring scheme that address the scale, material identity, behavior, and distribution of particles concerned with the scientific concept.
 20. A non-transitory computer-readable medium encoded with instructions for commanding one or more data processors to execute steps of a method of processing a drawing in a modeling prototype, the steps comprising: accessing a data structure associated with a visual model; analyzing the visual model to extract construct-relevant features, wherein the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn; analyzing the visual model to generate a statistical model, wherein the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs; determining an automated scoring based on the construct-relevant features and the statistical model, wherein the automated scoring is stored in a computer readable medium and is outputted for display on a graphical user interface, transmitted across a computer network, or printed. 